DOCUMENT RESUME 



ED 407 544 



CE 073 969 



TITLE 



INSTITUTION 

SPONS AGENCY 

PUB DATE 
NOTE 

CONTRACT 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Research on and Guidelines for Effective Use of Assessment 
Instruments and Strategies for Adult Learners Enrolled in 
Adult Basic and Literacy Education Programs. Final Report. 
Pennsylvania State Univ., University Park. Inst, for the 
Study of Adult Literacy. 

Pennsylvania State Dept, of Education, Harrisburg. Bureau of 
Adult Basic and Literacy Education. 

30 Jun 96 

26p . ; For a related guide, see CE 073 968. 

98-6006 

Reports - Research (143) 

MF01/PC02 Plus Postage. 

Adult Basic Education; Basic Skills; Educational Research; 
♦Evaluation Methods; Evaluation Problems; Evaluation 
Research; Evaluation Utilization; * Informal Assessment; 
♦Literacy Education; ^Standardized Tests; *Student 
Evaluation 



ABSTRACT 



The goal of this research project was to create a guide on 
the effective use of assessment instruments and methodologies, related 
resources, and guidelines for measuring adult learners' attainment of basic 
skills and competencies to document educational gains and demonstrate program 
quality. The project focused on confirming current use of assessment in adult 
basic and literacy education (ABLE) programs and conducting focus groups to 
determine perceived barriers to using standardized assessments. Ongoing 
consultation with COSMOS Corporation (the project contractor) confirmed ABLE 
programs' current use of assessment instruments and methodologies. The 
project team conducted a literature review to develop supplemental 
information on recently published, commercially available assessments and on 
current assessment practices. This research also included conducting focus 
groups with ABLE practitioners to identify concerns and issues relating to 
assessment practices. Based on the research, project staff developed a basic 
guide to the effective use of standardized and informal assessments. (Focus 
group questions are appended.) (YLB) 
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ABSTRACT 



Title: Research on and Guidelines for Effective Use of Assessment Instruments and 
Strategies for Adult Learners Enrolled in Adult Basic and Literacy Education 
Programs 

Project No.: 98-6606 Funding: $37,470 

Project Director: Eunice N. Askov Phone No.: (814)863-3777 

Contact Person: Barbara Van Horn Phone No.: (814)863-3777 

Agency Address: Institute for the Study of Adult Literacy, The Pennsylvania State 

University, 102 Rackley Building, University Park, PA 16802-3202 

Purpose: The Institute for the Study of Adult Literacy developed a basic guide to the 
effective use of assessments in ABLE programs. With Bureau approval, the originally 
planned statewide survey of ABLE programs’ use of assessments was eliminated. Rather, 
the project focused on 1) confirming current use of assessments in ABLE programs and 2) 
conducting focus groups to determine perceived barriers to using standardized 
assessments. The resulting guide focused on using standardized assessments to enhance 
program accountability, using standardized and informal assessments to guide instruction 
and measure student learning, working with ESL adults and adults with learning 
difficulties, and establishing a supportive testing environment Finally, the guide included 
references and a supplement, updating G. Jackson’s Measures for Adult Literacy 
Programs. ABLE providers as well as professional development center staff should find 
the guide useful as general background information and for professional development 
purposes. 

Procedures: Ongoing consultation with COSMOS Corporation (EQuAL Project 
contractor) confirmed ABLE programs’ current use of assessment instruments and 
methodologies (Objective 1). The project team conducted a literature review to develop 
supplemental information on recently published, commercially available assessments and 
on current assessment practices (Objective 2). This research also included conducting 
focus groups comprised of ABLE practitioners to identify concerns and issues relating to 
assessment practices (Objective 3). Based on the research, project staff developed a basic 
guide to the effective use of standardized and informal assessments (Objective 4). 

Summary of Findings: Research findings indicated that ABLE providers 1) generally 
do not believe that standardized assessments accurately measure student gains, 2) may not 
follow standardized test administration guidelines, and 3) generally do not use the results of 
standardized assessments but prefer informal instruments to guide instruction and measure 
progress. However, ABLE providers also expressed interest in guidelines for more 
accurately documenting learner progress. Reviewers concluded that the guide’s structure 
and content were clear and that it would be useful for both new and experienced ABLE 
instructional staff. 

Comments: The project’s original objectives were modified after the contract was 
awarded. These modifications necessitated changes to the project narrative and budget but 
did not significantly affect the final project outcome. 

Products: Assessment guide 

Descriptors: 

BEST COPY AVAILABLE 



Research on and Guidelines for Effective Use of Assessment 
Instruments and Strategies for Adult Learners Enrolled in ABLE 

Programs 

Introduction 

Purpose/Objectives 

The Institute for the Study of Adult Literacy (College of Education, The 
Pennsylvania State University) proposed to build on previous research to develop a basic 
guide on the effective use of standardized assessment instruments and methodologies and 
on the collection and use of assessment data to place learners, guide instruction, and 
measure changes in adult learners’ literacy skills. The guide. Assessment and Adult 
Learners: Getting the Most From Standardized and Informal Assessment Instruments, 
focuses on using standardized assessments to enhance program accountability, using 
standardized and informal assessments to guide instruction and measure student learning, 
working with English as a Second Language (ESL) adults and adult with learning 
difficulties and establishing a supportive testing environment. Finally, the guide includes 
references and a supplement updating G. Jackson’s Measures for Adult Literacy 
Programs. 

Time Frame 

The project was conducted between July 1, 1995 and June 30, 1996. 

Key Personnel 

Co-Principal Investigators Eunice N. Askov and Barbara Van Horn, with the 
assistance of Shawn Jenkins (graduate assistant) and Priscilla Carman (project associate), 
conducted the research and developed the guide. Dr. Askov and Ms. Van Horn directed 
the research, drafted the guide, and conducted dissemination activities. Ms. Van Horn 
also communicated with COSMOS Corporation and reviewers; supervised collection and 
summary of data (information on newly identified instruments and focus group results); 



collected and used evaluative data to revise drafts; coordinated dissemination efforts; and 
wrote the final report. Ms. Carman assisted Dr. Askov and Ms. Van Horn in designing 
and developing the guide. Ms. Jenkins assisted Ms. Van Horn in reviewing the literature 
for appropriate assessment instruments; contacting publishers for information on 
instruments; designing, conducting, and analyzing data from the focus groups; drafting 
segments of the guide; drafting the final report; and disseminating project results. A 
transcriber transcribed taped focus group data for analysis. 

The reviewers assisted project staff by providing feedback on drafts of the guide, 
identifying additional source and resources, and assisting with scheduling focus groups. 
Focus group participants attended one-hour focus group meetings to answer the targeted 
questions. 

Audience 

ABLE providers should find the guide useful as general background on 
assessment and for selecting and using basic skills assessments to meet the needs of 
various stakeholders. Administrators, counselors, and instructional staff have a resource 
to 1) guide their selection of assessments to meet the needs of various stakeholders, 2) 
provide ideas for screening adults for learning disabilities and offering accommodations 
when assessing adults with learning difficulties, 3) provide strategies for using 
standardized and informal assessments to guide instruction and measure learning, and 4) 
inform decisions related to indicators of program quality. Professional development 
center staff have a resource that can contribute to local and regional training opportunities 
on learner assessment. 

Dissemination 

Permanent copies of this report and the guide (including a copy of G. Jackson’s 
Measures for Adult Literacy Programs) are filed at the Bureau of Adult Basic and 
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Literacy Education and at the Commonwealth’s two literacy resource centers and six 
professional development centers. 



Addresses for the Bureau and literacy resource centers are: 

1. Bureau of Adult Basic and Literacy Education 
Market Street, 12th Floor 

Harrisburg, PA 17126-0333 

2. AdvancE 

Market Street, 1 1th Floor 
Harrisburg, PA 17126-0333 

3. Western PA Adult Literacy Resource Center 
William Flynn Highway, Route 8 
Gibsonia, PA 15044-9644 

Copies of the report and guide are also available through the Institute for the 
Study of Adult Literacy, Penn State University, 102 Rackley Building, University Park, 
PA 16802-3202. 
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Statement of the Problem 



Traditionally, adult basic and literacy education (ABLE) programs have depended 
on the results of standardized, norm-referenced tests to determine changes in participants’ 
basic skills levels and to report pre- and posttest data. The Bureau of ABLE (1994) 
encourages the use of standardized testing to conduct student assessments, reporting that 
44% of Pennsylvania’s ABLE providers use the Tests of Adult Basic Education — by far 
the most commonly used test. On the other hand, 46% of ABLE programs report using 
“other” assessments. The Bureau’s 1994 report does not list these other assessments; 
however, it is expected that they include both less frequently used standardized 
instruments as well as alternative and informal assessments. Fluke (1991) reports, in fact, 
that many programs use standardized tests only to “provide a launching point for 
designing a plan for teaching”. 

Many adult educators are generally dissatisfied with the quality of information 
derived from standardized, norm-referenced tests and with the effects of these tests on 
both teaching and learning (Ehringhaus, 1991; Lytle, 1988). Dissatisfied with the 
traditional norm-referenced standardized tests, instructors have relied increasingly on 
alternative assessments as more realistic and reliable measures of learner progress. 
Programs have begun to explore the use of alternative — and often informal — 
assessments; however, it is unlikely that the Bureau of ABLE will eliminate the use of 
standardized testing in favor of alternative assessments. The Bureau’s need to generate 
generalizable data on student progress that will encourage continued Federal and 
Commonwealth support for ABLE programs has increased dramatically in the past 
several years. Based on her direction of Pennsylvania’s EQuAL Project, J. Alamprese 
(personal communication, August 8, 1995) reported that providers’ inappropriate use of 
standardized instruments and subsequent collection of student progress data hamper the 
Commonwealth’s efforts to support and enhance program accountability. 
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While others have compiled lists of commercially available ABE, GED, and ESL 
tests (e.g.. Fluke, 1991; Jackson, 1990; Sticht, 1990), alternative assessment 
methodologies, such as portfolio assessment, have been mentioned only briefly, if at all. 
Today, the trend in student assessment appears to be toward supplementing standardized 
instruments with less structured alternative or informal assessment methods which hold 
particular promise for assessing adults with disabilities who may not perform well on 
standardized instruments. In response to instructors’ growing interest in alternative 
assessments, various models have emerged (e.g., Fingeret, 1993; Lazar & Bean, 1993; 
Lytle & Wolfe, 1989). To date, however, compilation and evaluation of these alternative 
assessments or guidelines for effectively using the instruments and methodologies do not 
exist 

Goals and Objectives 

The goal of the research was to create a guide on the effective use of assessment 
instruments and methodologies, related resources, and guidelines for measuring adult 
learners’ attainment of basic skills and competencies to document educational gains and 
demonstrate program quality. The objectives of the project were: 

1. To confirm current use of assessment instruments and methodologies used in ABLE 
programs through contact with COSMOS Corporation (as contractor responsible for 
the EQuAL project). 

2. To research and develop a supplement to existing information (e.g., Jackson) on 
recently published assessments appropriate for learners enrolled in ABLE programs. 

3. To conduct five focus groups comprised of representatives from programs serving 
adults enrolled in literacy, ABE, GED, and ESL programs, and adults with disabilities 
to identify concerns, interests, and/or barriers to using standardized, norm-referenced 
assessment instruments and methodologies. 
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4. To create a guide to the effective use of standardized, norm-referenced assessments 
that will support and enhance program accountability, including guidelines for using 
assessment results to guide instruction and measure student learning. 

Procedures 

The following section of the report details procedures used to meet the project 
objectives. 

1: Confirm current ABLE program use of assessment instruments and 
methodologies. 

Ms. Van Horn communicated with Ms. J. Alamprese, researcher with COSMOS 
Corporation and EQuAL project contractor, to identify standardized assessment 
instruments used in the EQuAL sites. This process updated Fluke’s research (1991) on 
ABLE providers’ use of various assessment instruments and drew project staff’s attention 
to newly published instruments that could be included in the guide. It also informed 
development of the guide by highlighting the need to provide essential information 
concerning assessment procedures (e.g., importance of following standardized 
assessment guidelines to obtain valid data), basic information concerning the purposes for 
assessment, and suggestions for selecting and administering various types of assessment 
instruments. 

2: Develop a supplement to existing information on recently published assessments. 

Project staff conducted a comprehensive review of the research and practice 
literature to identify recently published assessment instruments and information on 
assessment methods currently used in adult education programs for adult basic education 
(ABE), general education development (GED), and English as a second language (ESL), 
including those for adults with disabilities. These searches included informational 
resources available in the Institute’s library, Penn State’s Pattee Library, and the state 
literacy resource centers. In addition, staff searched various databases including the 



ERIC system and other electronically available resources (e.g., Gopher servers and World 
Wide Web). The Institute also queried relevant electronic discussion groups (i.e., 
listservs), such as the listservs sponsored by the National Institute for Literacy, for 
currently used assessment instruments and methods. Staff contacted adult educators 
conducting research and/or demonstrations of assessment strategies appropriate for adult 
learners (e.g., researchers and practitioners at the National Adult Literacy and Learning 
Disabilities Center, National Institute for Literacy, Literacy South, Northwest Regional 
Educational Laboratory) as well as publishers of adult education materials to identify 
alternative and/or non-commercial and commercial assessment instruments for adult 
literacy. 

Project staff summarized essential information about methods and newly 
published assessment instruments. Summaries of assessment instruments included name; 
source of availability; publication date; administration requirements; and technical 
information including reliability, validity, and norming group (if applicable). These 
summaries, which added to test descriptions in Gregg Jackson’s Measures for Adult 
Literacy Programs, were included as an appendix to the guide. A copy of Jackson’s 
book was included with copies of the guide sent to Pennsylvania’s professional 
development centers and state literacy resource centers. 

3: Conduct focus groups. 

Project staff developed focus group questions to determine ABLE providers’ 
concerns about the effective use of standardized, norm-referenced basic skills 
assessments. Data from the focus groups assisted project staff in designing the guide to 
address these concerns and issues, presenting a strong rationale for using standardized 
assessments effectively for various purposes. This information also was useful in 
formulating recommendations for the effective use of various instruments and methods. 



Focus Group Procedures 



Ms. Jenkins drafted the focus group questions. Rather than asking the reviewers 
(members of the PAACE Board of Directors) as originally intended, Dr. Debra Wilcox 
Johnson (a colleague of Ms. Van Horn’s with extensive experience in planning and 
conducting focus groups) agreed to review the draft questions and provide feedback. 

Ms. Jenkins also contacted selected EQuAL site administrators and the reviewers 
and requested assistance in identifying and recruiting focus group participants. She 
mailed information concerning the project and the focus groups (e.g., focus group 
purpose and questions, informed consent documents explaining how data would be used). 
When potential focus group participants were identified, Ms. Jenkins contacted these 
individuals to schedule their participation in one of the five groups. Only three focus 
groups were conducted due to unusually severe winter weather. A total of 36 participants 
met in focus groups during January, 1996 (9 participants in Pittsburgh, 4 in Scranton, and 
23 in Allentown) to explore assessment practices being used in Pennsylvania ABLE 
programs. These participants represented volunteer literacy, ABE, GED, ESL, and 
corrections and vocational education. They also represented various positions within 
ABLE programs: administrators, instructors, tutors, counselors, intake counselors, and 
test administrators. The groups responded to questions relating to types of tests, uses for 
tests, and testing preferences and concerns (See the Appendix for the list of focus group 
questions). Focus group discussions were audiotaped for later transcription. 

Data Analysis and Findings 

A transcriber, under the direction of project staff, transcribed the interview data 
from the audiotapes. Data analysis identified key issues and concerns about using 
standardized, norm-referenced basic skills assessments and methods. Ms. Jenkins used 
Hyper Qual software to facilitate coding and analysis of qualitative data. This process 
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allowed the comparison of data across focus groups to identify trends and strategies for 
designing the guide to address providers’ concerns. 



Data analysis revealed that providers were using assessments for various 
purposes. As expected, many different tests were being used; however, test 
administrators did not always adhere to the instructions for administering standardized 
instruments (e.g., ignoring time limits). Some of the tests that participants mentioned 
were standardized norm- or criterion-referenced instruments, developed for adults (e.g., 
ABLE, API, TABE, ALERT, BEST). Other assessments were curriculum-based 
assessments (e.g., Number Power and Challenger). However, several instruments were 
either designed and normed on children (e.g., Botel Reading Inventory, Spache) or were 
dated and out of print (e.g., Thumbnail Diagnostic Placement Test). Other measures 
generally were teacher-made, informal assessments appropriate for local program use. 
Many participants indicated that they depended on teacher intuition and informal 
methods such as portfolios to assess learner gains. The chart on the next page lists the 
assessments that focus group participants mentioned as commonly used in their 
programs. 
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Sample of Assessment Tools Used in Pennsylvania 



*■ *' Published Assessment Tools^ 


• — •♦♦♦«♦♦ «••••••••••••••••••• «•«•••••<• •«•••••••• ♦♦♦♦♦ •••• t ••• ••• • ♦••••#♦« 

' Unpublished :**?■■■. 


... . .. ,..* v. , •• ,. r . .... . . V 


... .... . . ... 


1. Adult Basic Learning Examination (ABLE) 


1. goal accomplishment lists 


2. Adult Placement Indicator (API) 


2. interest surveys 


3. ALERT 


3. journals 


4. APTICOM 


4. personal interviews 


5. Basic English Skills Test (BEST) 


5" pbrSbilos 


6. Bote! Reading Inventory 


6. progress reports 


7. Brigance 


7. questionnaires 


8. Challenger Series tests 


8. teacher-made tests 


9. ESL Oral Assessment (ESLOA) 


9. wffting samples 


10. Language Assessment "Scales (LAS) 




1 1. McGrath Test of Reading Skills 




12. Number Power 


1. Career Compass 


13. Scott-Foresman informal inventory ~ 


27~CDM (Career DecisionMaking) 


14. Slosson Oral Reading Rest (SORT) 


3. Hanington-O’Shay 


B. Spache 


4. Work Values Inventory 


16. Tests of Adult Basic Education (TABE) 


5. LD screening 


17. TABE Workplace Foundations 


6. MELT levels 


1 8 . Thumbnail Diagnostic Placement T est 


7. Seif-Directed Search 


19. Tutors of Literacy in the Commonwealth 
assessment (TLC) 


8. Indicators of frogram Quality 

(IPQ) 
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Assessment procedures . Focus group participants expressed many viewpoints; 
however, common assessment procedures emerged. Most providers conduct multiple 
levels of testing as students enter their programs. Usually, program staff conduct an 
initial interview, including a questionnaire to determine past educational history and 
personal goals related to entering the program. It should be noted, however, that 
representatives from ESL programs stated that their ABLE programs’ policies made it 
difficult, at best, to conduct effective initial testing. In these programs, many students 
with no or very little English proficiency are enrolled and assigned to a class. Instructors 
are expected to test these learners when they join the class. Unfortunately, these classes 
may be very large. As a result, instructors often do not have time to conduct even 
cursory screening, beginning instruction with no information on the adults’ English 
proficiency levels. Also as part of the intake process, some students are screened for 
learning disabilities. Some participants indicated that disability screening was part of the 
intake process; others screened learners at a later time. Some programs did not conduct 
disability screening. 

Following intake, providers generally follow two practices. In one case, staff 
administer a placement test to identify an approximate skill level so the student can be 
placed in an appropriate class. A more comprehensive standardized test is given later to 
determine an initial grade equivalent for administrative reporting requirements. In the 
second case, placement tests are not used. Program staff administer a comprehensive 
standardized test immediately, using test scores to place students. In some instances, a 
placement or locator test is given prior to the comprehensive test, but this is used only to 
determine which level of the test to administer. 

After initial testing, various diagnostic assessments are used to determine an 
appropriate instructional plan. These assessments included both published and teacher- 
made tests. Reading, writing, and math skills are usually evaluated. Students in 



11 



ERIC 




vocational programs also are given vocational aptitude testing followed by goal planning 
based on the student's interests and abilities. 



Schedules for posttesting varied widely. Some programs administered a posttest 
after 12 weeks of class, some after a predetermined number of contact hours, and others 
at the end of the program year. Providers often have students who could not be 
posttested because they had exited the program. Posttest results were used for program 
accountability and for measuring student progress. 

Assessment issues . Several concerns about assessment arose in the discussions. 
Not all standardized tests were administered in accordance with recommended practice, 
thus invalidating the learners’ scores. This usually referred to ignoring time limitations. 
Some providers related that students were allowed extra time, but they were only graded 
on the questions completed within the time allowances. 

A second concern was testing logistics. In most cases, students were asked to 
complete many levels of testing. Some participants felt that it was burdensome and 
discouraging for students, while others felt students expected the testing and accepted it 
Also, testing occasionally didn’t seem feasible due to the large enrollment of students at 
one time (such as at the first meeting of an ESL class). 

Some practitioners brought up a third concern. They questioned the need for 
specific testing or screening in some areas such as disabilities. Many practitioners felt 
that teacher intuition was particularly valid and was verified by test scores. Some 
practitioners felt that teacher intuition alone was sufficient to screen for potential 
disabilities or to place learners in appropriate instructional levels. 

A final concern expressed by most participants was the role of alternative 
assessment practices. Many of the focus group participants felt that alternative 
assessment methods such as portfolios, writing samples, checklists, goal sheets, and 
journals provided a better picture of the student’s ability than did standardized tests. The 
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participants saw the validity in using these assessment measures but acknowledged the 
need for providing data that can be generalized for accountability purposes. They felt the 
need to validate alternative assessment methods so that they would be acceptable to 
funding agencies for accountability. 

4: Create a guide. 

Project staff drafted a guide to include data collected and analyzed as part of the 
research conducted in Objectives 2—3. Efforts were made to address specific concerns 
that practitioners raised during the focus groups. For example, the guide addressed the 
importance of following instructions for administering standardized tests, the need for 
multiple assessments to meet various program objectives and information needs, the use 
of test accommodations for adults with learning difficulties, the use of assessment results 
to measure learner progress, and issues related to standardizing informal assessments. 

Originally, reviewers were to be drawn from the Pennsylvania Association for 
Adult Continuing Education (PAACE) program directors. However, as the project 
developed, the Bureau suggested that reviewers be drawn from Project EQuAL sites. As 
a result, four reviewers with expertise in adult literacy, ABE, GED, ESL, family literacy, 
and workplace literacy were identified. These reviewers as well as several individuals 
who attended the 1996 Commission on Adult Basic Education (CO ABE) conference 
session on the project read the draft, paying particular attention to their area(s) of 
expertise. EQuAL project staff also were helpful in reviewing drafts of the guide and 
providing feedback. Feedback from all reviewers was used to revise the draft. (See the 
Appendix for the review sheet) 

The guide provides an introduction to assessment methods and approaches 
appropriate for adult learners, defining purposes, types of assessments, and terms such as 
reliability and validity. Suggestions are included for selecting appropriate assessment 
instruments and providing a supportive testing environment In addition, the guide 



provides information on assessing adults with disabilities and those for whom English is 
a second language. The guide encourages ABLE providers to include adult learners in 
the assessment process, providing them with meaningful information about their 
progress. Finally, the guide will assist ABLE staff in using assessment results to guide 
instruction (placement and diagnosis) and to evaluate student learning. The appendix 
includes screening procedures designed by the National Adult Literacy and Learning 
Disabilities (NALLD) Center to identify adults with potential learning disabilities. It also 
includes summaries of identified assessment instruments, supplementing Jackson’s 
(1990) book on assessments for adult learners. [A copy of the Jackson book was 
included with guides sent to Pennsylvania’s professional development centers and 
literacy resource center.] 

Objectives Met/Not Met 

The Bureau’s requested revisions to the project’s plan required staff to refocus 
work from developing and conducting a statewide survey on assessment to designing and 
conducting focus groups. These procedural revisions were not finalized until September, 
1996. 

Objectives were satisfactorily met; however, most objectives experienced delays. 
For example, test publishers were not prompt in sending information about their 
assessments, particularly information related to reliability and validity. In some cases, 
information was never received. As a result, collecting and reviewing assessment 
instruments for inclusion in the supplement to Jackson’s book (Objective 2) was 
hampered. In addition, project staff conducted only three of the originally planned focus 
groups (Objective 3) due to an unusually severe winter. On the other hand, over 20 
individuals participated in the focus group in Allentown. As a result, project staff were 
still able to sample a representative cross-section of practitioners and administrators. 



Evaluation Instruments/T echniques 



Formative Evaluation and Recordkeeping 

Formative evaluation allowed project staff to refine their approach to the project 
and respond to potential barriers; therefore, formative measures were included in each 
objective. Project records documented progress toward each objective and assisted staff 
in revising the projected activities as necessary. The formative evaluation plan included 
internal quality control measures and external review. Dr. Askov and Ms. Van Horn 
communicated to ensure that timelines were met — or adjusted realistically in response 
to barriers — for each objective. 

1 : Confirm current ABLE program use of assessment instruments and methodologies. 

Project staff used information obtained from COSMOS Corporation to identify 
additional assessment instruments as well as issues and concerns identified by the 
EQuAL project about the use of standardized assessments. COSMOS’ information also 
was validated against information obtained via the focus groups. This information, 
gathered throughout the project period, informed the development of the guide. 

2: Develop a supplement to existing information on recently published assessments. 

Soliciting input via electronic networks ensured that the literature review and 
search for newly published instruments was comprehensive. Project staff also used 
reviewers’ suggestions to modify the supplement. 

3: Conduct focus groups. 

An expert on focus groups reviewed the draft focus questions; her suggestions 
were used for revisions. Staff review of the focus group participants indicated broad 
representation from various types of program providers. Data analysis revealed common 
concerns and issues that were subsequently addressed in the guide. 
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4: Create a guide. 



External reviewers ensured that the guide was understandable and usable. 
Reviewers’ written responses to the draft materials assisted the staff in revising the guide. 

Summative Evaluation 

Summative evaluation documents the project’s effect on the targeted population. 
It is difficult to collect summative data in a 12-month project; however, project staff 
collected data from COABE conference participants as well as other individuals who 
reviewed the guide prior to the end of the project These responses suggested that 
information in the guide would have a positive effect on practitioners’ knowledge, 
understanding, and use of assessment instruments and methodologies. For example, one 
conference participant stated, ’The idea that the results obtained from standardized tests 
are, in fact, valuable from a programming point of view and can provide information 
other than a grade level equivalent is encouraging. The do’s and don’t’s section for the 
use of standardized tests is really relevant since, as evidenced by comments from the 
small group of participants in the discussion, the ’rules’ do get abused or misconstrued.” 
A reviewer added, “I think it’s important for any educator to think seriously about how 
the kind of assessment relates to the purpose of that assessment.” 



Pennsylvania’s literacy resource centers received copies of the guide and final 
report. Professional staff development centers also were sent a copy of the guide; these 
copies included a copy of Jackson’s (1990) publication on assessment instruments. The 
Institute for the Study of Adult Literacy also will distribute the guide at cost. Project 
information and results will be included in the May, 1997 issue of the Institute’s Mosaic: 
Research Notes on Literacy (distributed nationally and internationally). 

Project staff presented information about the research and guide at the annual 
conference of the Commission of Adult Basic Education (COABE) in Pittsburgh (May, 
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1996). Ms. Van Horn presented an overview on assessment based on the project at the 
Summer Institute: New Teacher Orientation (August, 19%) and was scheduled to 
conduct a session on assessment at the national conference of the American Association 
of Adult and Continuing Education (October, 19%). 

Conclusions and Recommendations 

Conclusions 

The focus groups were useful in two ways. Participants assisted project staff by 
articulating current assessment practices as well as concerns and issues they have about 
assessment. They also identified some assessment tools that were listed as “other” in 
their programs’ annual report to Pennsylvania’s Bureau of Adult Basic and Literacy 
Education. 

Practitioners understood the need to document learner progress for accountability 
purposes but struggle to reconcile this need with the use of traditional standardized 
assessment instruments. For example, practitioners were concerned that data collected 
through standardized instruments does not adequately reflect learner progress. On the 
other hand, informal measures (e.g., portfolio assessment or locally developed 
instruments) did not provide the generalizable data needed for accountability purposes. 
These issues were mirrored in the literature review: 

1) Practitioners’ are increasingly interested in using informal learner-centered 

assessments as an alternative to standardized assessments for measuring 
learner progress. 

2) Funders’ challenge to ABLE programs for improved accountability (i.e., 

collecting generalizable data to document learner progress) is growing. 

Focus group participants and reviewers representing adult literacy programs 
highlighted the difficulties in assessing adults with learning difficulties (documented or 
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undocumented). Providers estimated that up to 50% of adults enrolled in literacy 
programs exhibit learning difficulties (e.g., specific learning disabilities, developmental 
delays, mental health problems, attention deficit disorder, brain trauma). Historically, 
documenting learner progress based on standardized norm-referenced tests (e.g., TABE) 
for this population has been problematic. With the increased emphasis on collecting data 
that can be aggregated (usually via standardized, norm-referenced tests), adult literacy 
programs were struggling to identify assessment tools appropriate for this population. 

Many ABLE instructors also requested information on using informal 
assessments: Are they valid? Are they reliable indicators of learner progress? Can 
informal instruments be standardized, thereby making data usable for accountability? 

Finally, research conducted in this project highlighted the need for ongoing 
professional development for adult educators as to the purposes for assessment, selection 
of appropriate instruments for each purpose, and procedures for administering 
standardized tests and recording data. Professional development efforts also should assist 
ABLE instructors in linking assessment more closely with curriculum and instruction. 
Additionally, the research suggested that providers also would benefit from assistance on 
implementing program modifications to improve the testing environment. Adult learners 
often are not prepared either psychologically or academically for testing. On the other 
hand, some programs do not provide instructors with adequate information about 
learners’ skills, hindering their ability to plan effective instruction. 

Recommendations 

1. Develop a series of professional development modules on assessment so that all 
ABLE practitioners become familiar with effective assessment strategies and 
procedures. It is essential that practitioners understand the importance of careful data 
collection and reporting for program evaluation and accountability since these 
practices affect both the individual program’s quality as well as the state’s ability to 




23 



18 



attract federal funds to support local programs. Practitioners also need assistance in 
linking assessment to curriculum and instruction. The guide developed through this 
project should be included as part of these modules. 

2 . Encourage programs to establish clear, appropriate policies and procedures 
concerning effective assessment practices as well as annual plans for 
training/retraining staff on these policies and procedures. Appropriate procedures 
might include, for example, ensuring that all students complete an intake interview 
and, at minimum, initial screening prior to being assigned to a class (impact on 
quality of instruction). Policies might include, for example, requirements that 
guidelines for using standardized assessment tools be exactly followed and any 
variations documented and explained (impact on quality of data). 

3. Research and design a process by which ABLE programs can report accurate and 
usable data on the progress of adults with learning difficulties. 

4. In a related issue, explore avenues for identifying and validating indicators that 
practitioners find compelling proof of learner progress (e.g., items included in 
informal assessments) and ways to capture usable data for program accountability, 
particularly for adults with learning difficulties and adults for whom English is a 
second language. 
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Focus Group Questions 



Warm up: Have participants introduce selves and briefly describe assessments they 
currently use in their programs. 

1. What forms of assessment are most useful to you? 

• What qualities make them useful? 

2. Several of you have mentioned the TABE and other standardized tests. How are these 
tests used in your program? 

• Changes in learners’ literacy levels (pre- and post-tests)? 

• Diagnosis of learners’ skills? 

• Placement? 

3. What are your questions or concerns about the standardized assessments you’ve 
described? 

• How do learners respond? 

4. How can standardized tests be used most effectively? 

5. How can State and Federal program accountability requirements and instructional 
concerns both be satisfied? 



6. What information about standardized tests would you like to know? 



REVIEW QUERIES 



Assessment and Adult Learners: Getting the Most from Standardized and 

Informal Assessment Instruments 

This guide is expected to provide ABLE providers with basic, yet essential, information 
about the effective and appropriate uses of both standardized and informal assessment 
instruments. The Bureau expects that the guide will complement work being done through 
Project EQuAL and will provide a basis for professional development for ABLE staff in the 
area of adult assessment 

Please consider the following aspects of the content and format of the guide. Please return 
your review comments to Barbara Van Horn via fax: 814-863-6108 or via the enclosed 
postage paid envelope. Thanks! 

Content 

1. Based on the outline, is the scope of the guide adequate? If not what would you add? 



2. Is the sequence of information logical and clear? If not, what would you change? 



3. Rough drafts of Chapters 1, 2, 4, 5, and 6 are attached. Based on these drafts, is the 
information included sufficient? If not, what would you add? 



4. Is too much information included? If so, what would you de-emphasize or delete? 



Style and Format 

Several Institute staff contributed to the enclosed draft chapters. Each has a particular style 
of writing, so the writing does not sound cohesive. Not to worry! We’ll resolve this in the 
final version. 

1. In general, are the chapters readable and accessible? Is the writing clear and 
straightforward? What do you like or dislike about the style/s of writing? 



2. Formats for the chapters vary. What do you prefer: one column or two? single or 
double spaced? indent at the beginning of a paragraph or no indent (for single spaced 
format)? 
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